{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow BYOM: Train with Custom Training Script, Compile with Neo, and Deploy on SageMaker\n",
    "\n",
    "This notebook can be compared to [TensorFlow MNIST distributed training notebook](https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker-python-sdk/tensorflow_distributed_mnist/tensorflow_distributed_mnist.ipynb) in terms of its functionality. We will do the same classification task, but this time we will compile the trained model using the Neo API backend, to optimize for our choice of hardware. Finally, we setup a real-time hosted endpoint in SageMaker for our compiled model using the Neo Deep Learning Runtime."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set up the environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sagemaker\n",
    "from sagemaker import get_execution_role\n",
    "\n",
    "sagemaker_session = sagemaker.Session()\n",
    "\n",
    "role = get_execution_role()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download the MNIST dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-2-3bec7422fb64>:5: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n",
      "WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please write your own downloading logic.\n",
      "WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use tf.data to implement this functionality.\n",
      "Extracting data/train-images-idx3-ubyte.gz\n",
      "WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use tf.data to implement this functionality.\n",
      "Extracting data/train-labels-idx1-ubyte.gz\n",
      "Extracting data/t10k-images-idx3-ubyte.gz\n",
      "Extracting data/t10k-labels-idx1-ubyte.gz\n",
      "WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:290: DataSet.__init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n",
      "Writing data/train.tfrecords\n",
      "WARNING:tensorflow:From /home/ec2-user/SageMaker/DSOAWS/amazon-sagemaker/neo/utils.py:29: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.\n",
      "\n",
      "Writing data/validation.tfrecords\n",
      "Writing data/test.tfrecords\n"
     ]
    }
   ],
   "source": [
    "import utils\n",
    "from tensorflow.contrib.learn.python.learn.datasets import mnist\n",
    "import tensorflow as tf\n",
    "\n",
    "data_sets = mnist.read_data_sets('data', dtype=tf.uint8, reshape=False, validation_size=5000)\n",
    "\n",
    "utils.convert_to(data_sets.train, 'train', 'data')\n",
    "utils.convert_to(data_sets.validation, 'validation', 'data')\n",
    "utils.convert_to(data_sets.test, 'test', 'data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Upload the data\n",
    "We use the ```sagemaker.Session.upload_data``` function to upload our datasets to an S3 location. The return value inputs identifies the location -- we will use this later when we start the training job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = sagemaker_session.upload_data(path='data', key_prefix='data/DEMO-mnist')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Construct a script for distributed training \n",
    "Here is the full code for the network model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "import os\n",
      "import tensorflow as tf\n",
      "from tensorflow.python.estimator.model_fn import ModeKeys as Modes\n",
      "\n",
      "INPUT_TENSOR_NAME = 'inputs'\n",
      "SIGNATURE_NAME = 'predictions'\n",
      "\n",
      "LEARNING_RATE = 0.001\n",
      "\n",
      "\n",
      "def model_fn(features, labels, mode, params):\n",
      "    # Input Layer\n",
      "    input_layer = tf.reshape(features[INPUT_TENSOR_NAME], [-1, 28, 28, 1])\n",
      "\n",
      "    # Convolutional Layer #1\n",
      "    conv1 = tf.layers.conv2d(\n",
      "        inputs=input_layer,\n",
      "        filters=32,\n",
      "        kernel_size=[5, 5],\n",
      "        padding='same',\n",
      "        activation=tf.nn.relu)\n",
      "\n",
      "    # Pooling Layer #1\n",
      "    pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[2, 2], strides=2)\n",
      "\n",
      "    # Convolutional Layer #2 and Pooling Layer #2\n",
      "    conv2 = tf.layers.conv2d(\n",
      "        inputs=pool1,\n",
      "        filters=64,\n",
      "        kernel_size=[5, 5],\n",
      "        padding='same',\n",
      "        activation=tf.nn.relu)\n",
      "    pool2 = tf.layers.max_pooling2d(inputs=conv2, pool_size=[2, 2], strides=2)\n",
      "\n",
      "    # Dense Layer\n",
      "    pool2_flat = tf.reshape(pool2, [-1, 7 * 7 * 64])\n",
      "    dense = tf.layers.dense(inputs=pool2_flat, units=1024, activation=tf.nn.relu)\n",
      "    dropout = tf.layers.dropout(\n",
      "        inputs=dense, rate=0.4, training=(mode == Modes.TRAIN))\n",
      "\n",
      "    # Logits Layer\n",
      "    logits = tf.layers.dense(inputs=dropout, units=10)\n",
      "\n",
      "    # Define operations\n",
      "    if mode in (Modes.PREDICT, Modes.EVAL):\n",
      "        predicted_indices = tf.argmax(input=logits, axis=1)\n",
      "        probabilities = tf.nn.softmax(logits, name='softmax_tensor')\n",
      "\n",
      "    if mode in (Modes.TRAIN, Modes.EVAL):\n",
      "        global_step = tf.train.get_or_create_global_step()\n",
      "        label_indices = tf.cast(labels, tf.int32)\n",
      "        loss = tf.losses.softmax_cross_entropy(\n",
      "            onehot_labels=tf.one_hot(label_indices, depth=10), logits=logits)\n",
      "        tf.summary.scalar('OptimizeLoss', loss)\n",
      "\n",
      "    if mode == Modes.PREDICT:\n",
      "        predictions = {\n",
      "            'classes': predicted_indices,\n",
      "            'probabilities': probabilities\n",
      "        }\n",
      "        export_outputs = {\n",
      "            SIGNATURE_NAME: tf.estimator.export.PredictOutput(predictions)\n",
      "        }\n",
      "        return tf.estimator.EstimatorSpec(\n",
      "            mode, predictions=predictions, export_outputs=export_outputs)\n",
      "\n",
      "    if mode == Modes.TRAIN:\n",
      "        optimizer = tf.train.AdamOptimizer(learning_rate=0.001)\n",
      "        train_op = optimizer.minimize(loss, global_step=global_step)\n",
      "        return tf.estimator.EstimatorSpec(mode, loss=loss, train_op=train_op)\n",
      "\n",
      "    if mode == Modes.EVAL:\n",
      "        eval_metric_ops = {\n",
      "            'accuracy': tf.metrics.accuracy(label_indices, predicted_indices)\n",
      "        }\n",
      "        return tf.estimator.EstimatorSpec(\n",
      "            mode, loss=loss, eval_metric_ops=eval_metric_ops)\n",
      "\n",
      "\n",
      "def serving_input_fn(params):\n",
      "    inputs = {INPUT_TENSOR_NAME: tf.placeholder(tf.float32, [None, 784])}\n",
      "    return tf.estimator.export.ServingInputReceiver(inputs, inputs)\n",
      "\n",
      "\n",
      "def read_and_decode(filename_queue):\n",
      "    reader = tf.TFRecordReader()\n",
      "    _, serialized_example = reader.read(filename_queue)\n",
      "\n",
      "    features = tf.parse_single_example(\n",
      "        serialized_example,\n",
      "        features={\n",
      "            'image_raw': tf.FixedLenFeature([], tf.string),\n",
      "            'label': tf.FixedLenFeature([], tf.int64),\n",
      "        })\n",
      "\n",
      "    image = tf.decode_raw(features['image_raw'], tf.uint8)\n",
      "    image.set_shape([784])\n",
      "    image = tf.cast(image, tf.float32) * (1. / 255)\n",
      "    label = tf.cast(features['label'], tf.int32)\n",
      "\n",
      "    return image, label\n",
      "\n",
      "\n",
      "def train_input_fn(training_dir, params):\n",
      "    return _input_fn(training_dir, 'train.tfrecords', batch_size=100)\n",
      "\n",
      "\n",
      "def eval_input_fn(training_dir, params):\n",
      "    return _input_fn(training_dir, 'test.tfrecords', batch_size=100)\n",
      "\n",
      "\n",
      "def _input_fn(training_dir, training_filename, batch_size=100):\n",
      "    test_file = os.path.join(training_dir, training_filename)\n",
      "    filename_queue = tf.train.string_input_producer([test_file])\n",
      "\n",
      "    image, label = read_and_decode(filename_queue)\n",
      "    images, labels = tf.train.batch(\n",
      "        [image, label], batch_size=batch_size,\n",
      "        capacity=1000 + 3 * batch_size)\n",
      "\n",
      "    return {INPUT_TENSOR_NAME: images}, labels\n",
      "\n",
      "def neo_preprocess(payload, content_type):\n",
      "    import logging\n",
      "    import numpy as np\n",
      "    import io\n",
      "\n",
      "    logging.info('Invoking user-defined pre-processing function')\n",
      "\n",
      "    if content_type != 'application/x-image' and content_type != 'application/vnd+python.numpy+binary':\n",
      "        raise RuntimeError('Content type must be application/x-image or application/vnd+python.numpy+binary')\n",
      "    \n",
      "    f = io.BytesIO(payload)\n",
      "    image = np.load(f)*255\n",
      "\n",
      "    return image\n",
      "\n",
      "### NOTE: this function cannot use MXNet\n",
      "def neo_postprocess(result):\n",
      "    import logging\n",
      "    import numpy as np\n",
      "    import json\n",
      "\n",
      "    logging.info('Invoking user-defined post-processing function')\n",
      "    \n",
      "    # Softmax (assumes batch size 1)\n",
      "    result = np.squeeze(result)\n",
      "    result_exp = np.exp(result - np.max(result))\n",
      "    result = result_exp / np.sum(result_exp)\n",
      "\n",
      "    response_body = json.dumps(result.tolist())\n",
      "    content_type = 'application/json'\n",
      "\n",
      "    return response_body, content_type\n"
     ]
    }
   ],
   "source": [
    "!cat 'mnist.py'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The script here is and adaptation of the [TensorFlow MNIST example](https://github.com/tensorflow/models/tree/master/official/mnist). It provides a ```model_fn(features, labels, mode)```, which is used for training, evaluation and inference. See [TensorFlow MNIST distributed training notebook](https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker-python-sdk/tensorflow_distributed_mnist/tensorflow_distributed_mnist.ipynb) for more details about the training script.\n",
    "\n",
    "At the end of the training script, there are two additional functions, to be used with Neo Deep Learning Runtime:\n",
    "* `neo_preprocess(payload, content_type)`: Function that takes in the payload and Content-Type of each incoming request and returns a NumPy array\n",
    "* `neo_postprocess(result)`: Function that takes the prediction results produced by Deep Learining Runtime and returns the response body"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a training job using the sagemaker.TensorFlow estimator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "tensorflow py2 container will be deprecated soon.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-04 21:53:01 Starting - Starting the training job...\n",
      "2020-02-04 21:53:03 Starting - Launching requested ML instances......\n",
      "2020-02-04 21:54:08 Starting - Preparing the instances for training...\n",
      "2020-02-04 21:55:01 Downloading - Downloading input data\n",
      "2020-02-04 21:55:01 Training - Downloading the training image..\u001b[34m2020-02-04 21:55:13,519 INFO - root - running container entrypoint\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:13,519 INFO - root - starting train task\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:13,532 INFO - container_support.training - Training starting\u001b[0m\n",
      "\u001b[34mDownloading s3://sagemaker-us-east-1-806570384721/sagemaker-tensorflow-2020-02-04-21-53-00-662/source/sourcedir.tar.gz to /tmp/script.tar.gz\u001b[0m\n",
      "\n",
      "2020-02-04 21:55:13 Training - Training image download completed. Training in progress.\u001b[34m2020-02-04 21:55:31,589 INFO - tf_container - ----------------------TF_CONFIG--------------------------\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:31,589 INFO - tf_container - {\"environment\": \"cloud\", \"cluster\": {\"master\": [\"algo-1:2222\"]}, \"task\": {\"index\": 0, \"type\": \"master\"}}\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:31,590 INFO - tf_container - ---------------------------------------------------------\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:31,590 INFO - tf_container - creating RunConfig:\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:31,590 INFO - tf_container - {'save_checkpoints_secs': 300}\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:31,590 INFO - tensorflow - TF_CONFIG environment variable: {u'environment': u'cloud', u'cluster': {u'master': [u'algo-1:2222']}, u'task': {u'index': 0, u'type': u'master'}}\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:31,590 INFO - tf_container - creating an estimator from the user-provided model_fn\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:31,590 INFO - tensorflow - Using config: {'_save_checkpoints_secs': 300, '_keep_checkpoint_max': 5, '_task_type': u'master', '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f1887637190>, '_keep_checkpoint_every_n_hours': 10000, '_service': None, '_num_ps_replicas': 0, '_tf_random_seed': None, '_device_fn': None, '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_evaluation_master': '', '_eval_distribute': None, '_train_distribute': None, '_session_config': device_filters: \"/job:ps\"\u001b[0m\n",
      "\u001b[34mdevice_filters: \"/job:master\"\u001b[0m\n",
      "\u001b[34mallow_soft_placement: true\u001b[0m\n",
      "\u001b[34mgraph_options {\n",
      "  rewrite_options {\n",
      "    meta_optimizer_iterations: ONE\n",
      "  }\u001b[0m\n",
      "\u001b[34m}\u001b[0m\n",
      "\u001b[34m, '_global_id_in_cluster': 0, '_is_chief': True, '_protocol': None, '_save_checkpoints_steps': None, '_experimental_distribute': None, '_save_summary_steps': 100, '_model_dir': u's3://sagemaker-us-east-1-806570384721/sagemaker-tensorflow-2020-02-04-21-53-00-662/checkpoints', '_master': ''}\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:31,591 INFO - tensorflow - Skip starting Tensorflow server as there is only one node in the cluster.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:31.672504: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:31.673298: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:31,815 WARNING - tensorflow - From /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/input.py:187: __init__ (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.\u001b[0m\n",
      "\u001b[34mInstructions for updating:\u001b[0m\n",
      "\u001b[34mTo construct input pipelines, use the `tf.data` module.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:31,817 WARNING - tensorflow - From /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/input.py:187: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.\u001b[0m\n",
      "\u001b[34mInstructions for updating:\u001b[0m\n",
      "\u001b[34mTo construct input pipelines, use the `tf.data` module.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:31,840 INFO - tensorflow - Calling model_fn.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:32,140 INFO - tensorflow - Done calling model_fn.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:32,141 INFO - tensorflow - Create CheckpointSaverHook.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:32.153797: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:32.153880: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:32.175199: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:32.175227: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:32.229120: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:32.229190: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:32,568 INFO - tensorflow - Graph was finalized.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:32.577201: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:32.577234: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:32,968 INFO - tensorflow - Running local_init_op.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:32,973 INFO - tensorflow - Done running local_init_op.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:32,996 WARNING - tensorflow - From /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py:804: start_queue_runners (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.\u001b[0m\n",
      "\u001b[34mInstructions for updating:\u001b[0m\n",
      "\u001b[34mTo construct input pipelines, use the `tf.data` module.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:33.183536: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:33.183582: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:33,575 INFO - tensorflow - Saving checkpoints for 0 into s3://sagemaker-us-east-1-806570384721/sagemaker-tensorflow-2020-02-04-21-53-00-662/checkpoints/model.ckpt.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:35.959442: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:35.959489: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:36,967 INFO - tensorflow - loss = 2.2995076, step = 1\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:49,280 INFO - tensorflow - global_step/sec: 8.12105\u001b[0m\n",
      "\u001b[34m2020-02-04 21:55:49,281 INFO - tensorflow - loss = 0.08378975, step = 101 (12.314 sec)\u001b[0m\n",
      "\u001b[34m2020-02-04 21:56:01,595 INFO - tensorflow - global_step/sec: 8.12001\u001b[0m\n",
      "\u001b[34m2020-02-04 21:56:01,596 INFO - tensorflow - loss = 0.10527095, step = 201 (12.315 sec)\u001b[0m\n",
      "\u001b[34m2020-02-04 21:56:13,778 INFO - tensorflow - global_step/sec: 8.20827\u001b[0m\n",
      "\u001b[34m2020-02-04 21:56:13,779 INFO - tensorflow - loss = 0.07701896, step = 301 (12.183 sec)\u001b[0m\n",
      "\u001b[34m2020-02-04 21:56:25,885 INFO - tensorflow - global_step/sec: 8.25969\u001b[0m\n",
      "\u001b[34m2020-02-04 21:56:25,886 INFO - tensorflow - loss = 0.06647506, step = 401 (12.107 sec)\u001b[0m\n",
      "\u001b[34m2020-02-04 21:56:37,930 INFO - tensorflow - global_step/sec: 8.30218\u001b[0m\n",
      "\u001b[34m2020-02-04 21:56:37,931 INFO - tensorflow - loss = 0.018495549, step = 501 (12.045 sec)\u001b[0m\n",
      "\u001b[34m2020-02-04 21:56:50,217 INFO - tensorflow - global_step/sec: 8.13881\u001b[0m\n",
      "\u001b[34m2020-02-04 21:56:50,218 INFO - tensorflow - loss = 0.024498079, step = 601 (12.287 sec)\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:02,417 INFO - tensorflow - global_step/sec: 8.19673\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:02,418 INFO - tensorflow - loss = 0.055505965, step = 701 (12.200 sec)\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:14,527 INFO - tensorflow - global_step/sec: 8.25776\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:14,528 INFO - tensorflow - loss = 0.07919439, step = 801 (12.110 sec)\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:26,619 INFO - tensorflow - global_step/sec: 8.26951\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:26,620 INFO - tensorflow - loss = 0.027821526, step = 901 (12.093 sec)\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:38,703 INFO - tensorflow - Saving checkpoints for 1000 into s3://sagemaker-us-east-1-806570384721/sagemaker-tensorflow-2020-02-04-21-53-00-662/checkpoints/model.ckpt.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:41.236479: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:41.236568: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:41,757 INFO - tensorflow - Calling model_fn.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:41,931 INFO - tensorflow - Done calling model_fn.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:41,950 INFO - tensorflow - Starting evaluation at 2020-02-04-21:57:41\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:42,021 INFO - tensorflow - Graph was finalized.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:42,086 INFO - tensorflow - Restoring parameters from s3://sagemaker-us-east-1-806570384721/sagemaker-tensorflow-2020-02-04-21-53-00-662/checkpoints/model.ckpt-1000\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:42,547 INFO - tensorflow - Running local_init_op.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:42,555 INFO - tensorflow - Done running local_init_op.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:43,188 INFO - tensorflow - Evaluation [10/100]\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:43,649 INFO - tensorflow - Evaluation [20/100]\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:44,112 INFO - tensorflow - Evaluation [30/100]\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:44,580 INFO - tensorflow - Evaluation [40/100]\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:45,042 INFO - tensorflow - Evaluation [50/100]\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:45,510 INFO - tensorflow - Evaluation [60/100]\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:45,995 INFO - tensorflow - Evaluation [70/100]\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:46,467 INFO - tensorflow - Evaluation [80/100]\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:46,933 INFO - tensorflow - Evaluation [90/100]\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:47,394 INFO - tensorflow - Evaluation [100/100]\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:47,422 INFO - tensorflow - Finished evaluation at 2020-02-04-21:57:47\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:47,422 INFO - tensorflow - Saving dict for global step 1000: accuracy = 0.988, global_step = 1000, loss = 0.035483647\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:47.433921: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:47.433987: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:47.461659: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:47.461687: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:47.671599: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:47.671646: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:48,015 INFO - tensorflow - Saving 'checkpoint_path' summary for global step 1000: s3://sagemaker-us-east-1-806570384721/sagemaker-tensorflow-2020-02-04-21-53-00-662/checkpoints/model.ckpt-1000\u001b[0m\n",
      "\n",
      "2020-02-04 21:57:56 Uploading - Uploading generated training model\u001b[34m2020-02-04 21:57:48.201978: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:48.202014: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:48.265246: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:48.265281: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:48.285449: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:48.285477: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:48.306680: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:48.306713: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:48.330871: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:48.330931: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:48.350569: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:48.350632: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:48,444 INFO - tensorflow - Calling model_fn.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:48,502 INFO - tensorflow - Done calling model_fn.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:48,502 INFO - tensorflow - Signatures INCLUDED in export for Eval: None\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:48,502 INFO - tensorflow - Signatures INCLUDED in export for Classify: None\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:48,503 INFO - tensorflow - Signatures INCLUDED in export for Regress: None\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:48,503 INFO - tensorflow - Signatures INCLUDED in export for Predict: ['serving_default', 'predictions']\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:48,503 INFO - tensorflow - Signatures INCLUDED in export for Train: None\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:48,632 INFO - tensorflow - Restoring parameters from s3://sagemaker-us-east-1-806570384721/sagemaker-tensorflow-2020-02-04-21-53-00-662/checkpoints/model.ckpt-1000\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:49,000 WARNING - tensorflow - From /usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py:1018: calling add_meta_graph_and_variables (from tensorflow.python.saved_model.builder_impl) with legacy_init_op is deprecated and will be removed in a future version.\u001b[0m\n",
      "\u001b[34mInstructions for updating:\u001b[0m\n",
      "\u001b[34mPass your op to the equivalent parameter main_op instead.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:49,000 INFO - tensorflow - Assets added to graph.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:49,000 INFO - tensorflow - No assets to write.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:49.033535: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:49.033567: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:49.061441: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:49.061476: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:49.095429: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:49.095457: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:50.628731: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:50.628801: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:50,710 INFO - tensorflow - SavedModel written to: s3://sagemaker-us-east-1-806570384721/sagemaker-tensorflow-2020-02-04-21-53-00-662/checkpoints/export/Servo/temp-1580853468/saved_model.pb\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:50.720261: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:50.720295: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:52.009931: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:52.009972: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:52.035964: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:52.036029: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:52,085 INFO - tensorflow - Loss for final step: 0.085139.\u001b[0m\n",
      "\u001b[34m2020-02-04 21:57:52,380 INFO - tf_container - Downloaded saved model at /opt/ml/model/export/Servo/1580853468\u001b[0m\n",
      "\n",
      "2020-02-04 21:58:13 Completed - Training job completed\n",
      "Training seconds: 199\n",
      "Billable seconds: 199\n"
     ]
    }
   ],
   "source": [
    "from sagemaker.tensorflow import TensorFlow\n",
    "\n",
    "mnist_estimator = TensorFlow(entry_point='mnist.py',\n",
    "                             role=role,\n",
    "                             framework_version='1.11.0',\n",
    "                             training_steps=1000, \n",
    "                             evaluation_steps=100,\n",
    "                             train_instance_count=1,\n",
    "                             train_instance_type='ml.c4.xlarge')\n",
    "\n",
    "mnist_estimator.fit(inputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **```fit```** method will create a training job in two **ml.c4.xlarge** instances. The logs above will show the instances doing training, evaluation, and incrementing the number of **training steps**. \n",
    "\n",
    "In the end of the training, the training job will generate a saved model for TF serving."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deploy the trained model to prepare for predictions (the old way)\n",
    "\n",
    "The deploy() method creates an endpoint which serves prediction requests in real-time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The Python 2 tensorflow images will be soon deprecated and may not be supported for newer upcoming versions of the tensorflow images.\n",
      "Please set the argument \"py_version='py3'\" to use the Python 3 tensorflow image.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------------------------------------------------------------------------------------------!"
     ]
    }
   ],
   "source": [
    "mnist_predictor = mnist_estimator.deploy(initial_instance_count=1,\n",
    "                                         instance_type='ml.m4.xlarge')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Invoking the endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:252: _internal_retry.<locals>.wrap.<locals>.wrapped_fn (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use urllib or similar directly.\n",
      "Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.\n",
      "Extracting /tmp/data/train-images-idx3-ubyte.gz\n",
      "Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.\n",
      "Extracting /tmp/data/train-labels-idx1-ubyte.gz\n",
      "WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:110: dense_to_one_hot (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use tf.one_hot on tensors.\n",
      "Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.\n",
      "Extracting /tmp/data/t10k-images-idx3-ubyte.gz\n",
      "Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.\n",
      "Extracting /tmp/data/t10k-labels-idx1-ubyte.gz\n",
      "========================================\n",
      "label is 7\n",
      "prediction is 7\n",
      "========================================\n",
      "label is 2\n",
      "prediction is 2\n",
      "========================================\n",
      "label is 1\n",
      "prediction is 1\n",
      "========================================\n",
      "label is 0\n",
      "prediction is 0\n",
      "========================================\n",
      "label is 4\n",
      "prediction is 4\n",
      "========================================\n",
      "label is 1\n",
      "prediction is 1\n",
      "========================================\n",
      "label is 4\n",
      "prediction is 4\n",
      "========================================\n",
      "label is 9\n",
      "prediction is 9\n",
      "========================================\n",
      "label is 5\n",
      "prediction is 5\n",
      "========================================\n",
      "label is 9\n",
      "prediction is 9\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "\n",
    "mnist = input_data.read_data_sets(\"/tmp/data/\", one_hot=True)\n",
    "\n",
    "for i in range(10):\n",
    "    data = mnist.test.images[i].tolist()\n",
    "    tensor_proto = tf.make_tensor_proto(values=np.asarray(data), shape=[1, len(data)], dtype=tf.float32)\n",
    "    predict_response = mnist_predictor.predict(tensor_proto)\n",
    "    \n",
    "    print(\"========================================\")\n",
    "    label = np.argmax(mnist.test.labels[i])\n",
    "    print(\"label is {}\".format(label))\n",
    "    prediction = predict_response['outputs']['classes']['int64_val'][0]\n",
    "    print(\"prediction is {}\".format(prediction))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deleting the endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "sagemaker.Session().delete_endpoint(mnist_predictor.endpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deploy the trained model using Neo\n",
    "\n",
    "Now the model is ready to be compiled by Neo to be optimized for our hardware of choice. We are using the  ``TensorFlowEstimator.compile_model`` method to do this. For this example, our target hardware is ``'ml_c5'``. You can changed these to other supported target hardware if you prefer.\n",
    "\n",
    "## Compiling the model\n",
    "The ``input_shape`` is the definition for the model's input tensor and ``output_path`` is where the compiled model will be stored in S3. **Important. If the following command result in a permission error, scroll up and locate the value of execution role returned by `get_execution_role()`. The role must have access to the S3 bucket specified in ``output_path``.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The Python 2 tensorflow images will be soon deprecated and may not be supported for newer upcoming versions of the tensorflow images.\n",
      "Please set the argument \"py_version='py3'\" to use the Python 3 tensorflow image.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "?...........!"
     ]
    }
   ],
   "source": [
    "output_path = '/'.join(mnist_estimator.output_path.split('/')[:-1])\n",
    "optimized_estimator = mnist_estimator.compile_model(target_instance_family='ml_c5', \n",
    "                              input_shape={'data':[1, 784]},  # Batch size 1, 3 channels, 224x224 Images.\n",
    "                              output_path=output_path,\n",
    "                              framework='tensorflow', framework_version='1.11.0')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploying the compiled model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------------------------------------------------------------------------!"
     ]
    }
   ],
   "source": [
    "optimized_predictor = optimized_estimator.deploy(initial_instance_count = 1,\n",
    "                                                 instance_type = 'ml.c5.4xlarge')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def numpy_bytes_serializer(data):\n",
    "    f = io.BytesIO()\n",
    "    np.save(f, data)\n",
    "    f.seek(0)\n",
    "    return f.read()\n",
    "\n",
    "optimized_predictor.content_type = 'application/vnd+python.numpy+binary'\n",
    "optimized_predictor.serializer = numpy_bytes_serializer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Invoking the endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting /tmp/data/train-images-idx3-ubyte.gz\n",
      "Extracting /tmp/data/train-labels-idx1-ubyte.gz\n",
      "Extracting /tmp/data/t10k-images-idx3-ubyte.gz\n",
      "Extracting /tmp/data/t10k-labels-idx1-ubyte.gz\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAAAxUlEQVR4nGNgGDaAEUKFpD77sfTFHeyS9xQYGBg+X4UKPuk6w8DAwMDAAuGm6l/TMnSweCzLwPDntSTDozPIOhkYGBgYBA3PmDIw/Lh1XShnGi5nBP+9KIRLTuzl/2AokwlDMlv0/U1cGq1//rPDJcfQ+m83Ky45zrM/rHBqrPu3Daec9+8PlrjkhO/+W4ZLjvn0v9vKuCTV/v3zxSUn/+BfMSMuydZ//0xwydl+QpdEClsbHoa7X1AkWZA5F53f4TIWEwAAaRE8kJuHrgAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<PIL.Image.Image image mode=L size=28x28 at 0x7F1C59B474E0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========================================\n",
      "label is 7\n",
      "prediction is [0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.23196931183338165, 0.08533674478530884, 0.08533674478530884]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAAA6ElEQVR4nGNgoAlgRDBLOPVCGKYfX4xN2cq/f//+/fv3lhwOuat9G/7+rcKUM/n195ICDwPbub89mJK+vy9JMjAwVP3464jFWHkhBgYGhot/sUoyMDAwMJR+/3uMC4ecz/e/z+2R+EwormJjWHkQh8YN3/7O58EhJ/nq70tlXK459vdvLy45vx9/9+IyVPgEHo1tf/+uxaWR4cffv5LoYixIbKHfDAwMH3+z8jMIFjIw/C3/hix5iYGBgWH1c/FwCPdFKzwlrPNHqPrzj2HTGYYjxxHJpIyVgUE7nIFh3gOGdddxuWyAAQCfcVM+FkfDOQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<PIL.Image.Image image mode=L size=28x28 at 0x7F1C5AE37160>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========================================\n",
      "label is 2\n",
      "prediction is [0.08533674478530884, 0.08533674478530884, 0.23196931183338165, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAAAiElEQVR4nGNgGARA7V8unM2ELmn47ylune0fccvpfpmG4KAbq861ErfOU/e5ccop/LuBxEMz1p7hNW5JXYYunKZavj3LgVOns9CNHzgl9f+vwWmqxIvrKHwUnQliJ3BLyjO8x2kqw5N/Tjh12orj1sfQ++8sMy6dXF4Ma/7i0sh6bAMXHnPpBAAPgx/ARH1j7wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<PIL.Image.Image image mode=L size=28x28 at 0x7F1C59B474E0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========================================\n",
      "label is 1\n",
      "prediction is [0.08533674478530884, 0.23196931183338165, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAAA70lEQVR4nMXQsWoCQRQF0JvBNNrGdjcgIR8gJJVrqRZCfkMkgn+QlEIIAcHaHzClVWySJkmXSgxqIbKCbcrLxRSbNe7M2uqr7syZefAecOTK9fTp78MLUs2ds9nJ+b71OPMfWzdXAALz9ZrSVCQpclp0bbiRpPVckmPlmUh268Ed2bDsfEVx2skCfsif9qkzxcsZAOCWYsHGDy+K/nuM2zmNuV5E6cQYc5/4+UDG0W07iTFfXlGhl45PJGelKGeQrOElgPFb8vJbqtWW0kYpG2qT8W7ZtdEP/zAcFbI2IniMsOkIAKD6zEGl6qXjweoXXfV/5XmKZEMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<PIL.Image.Image image mode=L size=28x28 at 0x7F1C59B47550>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========================================\n",
      "label is 0\n",
      "prediction is [0.23196931183338165, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAAA2klEQVR4nGNgGLzA6AGU4SYLZTAhJN3ZoQy/bgxJFi8Y64wWN7qko+V8KEtIiwvNRt03N3mgzAN/RNEkV3w3hWn8/xdNMuTTZRiz9+9eVlTJlX+yoCyFF7+cUOX4H/6BMdv+wM2AupZdegVMRJnhCppzOM9cFIKwxP7+zYaJskCo73eDt/YxMDDoKMv/Z/iPppNBc9XXP3/+/Hnx/PefP5wwQUa4tKEyAwPDGoaF0TDTsID6P3900exEAEZGhss4Jf8jOYcJXZKD4QdOKxlevMnHLbnZCbcclQAA/k48Hcv/z+EAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<PIL.Image.Image image mode=L size=28x28 at 0x7F1C5AD07F98>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========================================\n",
      "label is 4\n",
      "prediction is [0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.23196931183338165, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAAAnElEQVR4nGNgGOzA9182My454cf//nHikgz8928pIw459tP//nni0mj6798vBI8JzVQGhl24NDIc+ffDAJec1b9/b5G4qMaaMjBMx2nq4n/vZHDJ2fz5dx+Zj2KsMBPDbnymmuKSk/nz7xKKALKxVkwMG3GamvnvlQhOnW4Mjz7ikmRVYfjxG5fkv9MMd1DtYUEw/9b8P4fTPdQEAJbDL46GK5NFAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<PIL.Image.Image image mode=L size=28x28 at 0x7F1C59B47550>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========================================\n",
      "label is 1\n",
      "prediction is [0.08533674478530884, 0.23196931183338165, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAAA10lEQVR4nGNgGLRA7ECbAozN78uKIif4+tdKuNydj6rIciJ7/06Gc7r/pqBodPv7VxTG1v63lhfFwpl/E+Byz//FoGhc/P8MN4yd8W8eqlMX/d0EdR9n89u/MFEWGMN714fpDAwM9g4WDGtQNTIYP/n799/fv3///vv797Yyms6zugYepa8XMjAsvshw7C4DDqD075woLjmGBX9dccqF/vtohFNy3r+lOOUYnn/BrTHj3wvcGi/8ncvAKwfnMqFJ/43e34xT57+/s2RxSNruaxBnw20rlQAAKNJLfTqR0FsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<PIL.Image.Image image mode=L size=28x28 at 0x7F1C59B474E0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========================================\n",
      "label is 4\n",
      "prediction is [0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.23196931183338165, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAAA1klEQVR4nGNgGKqAb+pBVlxy0Q/+/hXGISfz+t/fv8uEsEtO+Pvv79+/74rZsMjJf/x7Yeffv3+fS2CR9P93kIEj6fa//ycRJjPBGOz/+xl+zLv9//+3X5iSkQzeDAwMJgwMJ75gGhv294JG6LLfb/+90cKUFHr399/fvztVbvydgcVFLh/+/5vIwdD2774yNtl5fTwMDJzr/y7EIgkFEX8f4QgmBgYGpmV/63BrNfj6Vw23bPG/NZw4JUVv/dPDrVXu31Lckgy7vmAJJhjgu++HRysRAAA+/lIBnbxrFgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<PIL.Image.Image image mode=L size=28x28 at 0x7F1C5ADA7390>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========================================\n",
      "label is 9\n",
      "prediction is [0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.23196931183338165]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAAA6ElEQVR4nGNgoD9gxBARlGNgeFh45dZFTMXeM2/+/fv3+re/f9FllPu+/PkLAwwMDCzIkjL5EPrGVQgNlxQpOLLj18ev3LuunDz//Suqidzn/voxMCgwyDFhuoNt498WLhwe4mn9+5IfhxxDzN/7MliEIVZYMZx/gksjw6u/3+oNMYUhwff/HwPDvxkn5O5cZdA+jm5G918EeLECTZLZ9Na93zDZPzWYFjh7nIDKrsfmrIq/P2cYL8EhafT37989f/7+nYJNknP5379///5az41NkkF8y/O/dxuwSjEwMDDEThXDKUcfAAAG83bQTLLiMgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<PIL.Image.Image image mode=L size=28x28 at 0x7F1C59BA3860>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========================================\n",
      "label is 5\n",
      "prediction is [0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.23196931183338165, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAABBElEQVR4nN3QsS8DcRjG8UcOPUFiINLNYGgj2M7QpTGISSIxWRkMFomIGESCwWJ1XTtYDP4Ag60xoqNBqjW0IWlPQ/Tc9wyWa/P7/QOe7c3nfYb3lf5N+rqm6fHVPH7pybA46zcA6JQvBntortCE6uVpeMdrZavbCg24OXel20zp8zmaSJB7GFE/Gpakx5klIInLH9Q8SXKmdisBcXFAkvr/0IkULqxl9JXNvk1K9ZMw0Ry6bkcxhAD8XKV77hzbz72/pOY9Sf5B0/iTIrQ2HCNprwPrZtJmAOWU2bwWBDlL8RjaeYuNfoNvsZEa3LsWXIlh0WJ6gDObqRrX01bcibetZsovhERycinB3ycAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<PIL.Image.Image image mode=L size=28x28 at 0x7F1C59B47550>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========================================\n",
      "label is 9\n",
      "prediction is [0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.08533674478530884, 0.23196931183338165]\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "from IPython import display\n",
    "import PIL.Image\n",
    "import io\n",
    "\n",
    "mnist = input_data.read_data_sets(\"/tmp/data/\", one_hot=True)\n",
    "\n",
    "for i in range(10):\n",
    "    data = mnist.test.images[i]\n",
    "    # Display image\n",
    "    im = PIL.Image.fromarray(data.reshape((28,28))*255).convert('L')\n",
    "    display.display(im)\n",
    "    # Invoke endpoint with image\n",
    "    predict_response = optimized_predictor.predict(data)\n",
    "    \n",
    "    print(\"========================================\")\n",
    "    label = np.argmax(mnist.test.labels[i])\n",
    "    print(\"label is {}\".format(label))\n",
    "    prediction = predict_response\n",
    "    print(\"prediction is {}\".format(prediction))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deleting endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "sagemaker.Session().delete_endpoint(optimized_predictor.endpoint)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_tensorflow_p36",
   "language": "python",
   "name": "conda_tensorflow_p36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
